Virtualization, iSCSI, PCI Express and i-PCI
There is growing acceptance of techniques that leverage networked connectivity for extending the resources of host computer systems. In particular, networked connectivity is being widely utilized for specialized applications such as attaching storage to computers. For example, iSCSI makes use of TCP/IP as a transport for the SCSI parallel bus to enable low cost remote centralization of storage.
PCI Express, as the successor to PCI bus, has moved to the forefront as the predominant local host bus for computer system motherboard architectures. PCI Express allows memory-mapped expansion of a computer. A cabled version of PCI Express allows for high performance directly attached bus expansion via docks or expansion chassis.
A hardware/software system and method that collectively enables virtualization and extension of its memory map via the Internet, LANs, WANs, and WPANs is described in copending commonly assigned U.S. patent application Ser. No. 12/148,712 and designated “i-PCI”, the teachings of which are incorporated herein by reference.
The i-PCI protocol extends the PCI I/O System via encapsulation of PCI Express packets within network routing and transport layers and Ethernet packets and then utilizes the network as a transport. The network is made transparent to the host and thus the remote I/O appears to the host system as an integral part of the local PCI system architecture. The result is a virtualization of the host PCI System. The i-PCI protocol allows certain hardware devices (in particular I/O devices) native to the host architecture (including bridges, I/O controllers, and I/O cards) to be located remotely. FIG. 1 shows a detailed functional block diagram of a typical host system [105] connected to multiple remote I/O chassis [104]. An i-PCI host bus adapter card [101] installed in a host PCI Express slot [102] interfaces the host to the network. An i-PCI remote bus adapter card [103] interfaces the remote PCI Express bus resources to the network. For further in-depth discussion of the i-PCI protocol see U.S. patent application Ser. No. 12/148,712, the teachings which are incorporated by reference.
MeMAN
With the summary of iSCSI, PCI Express, i-PCI as a backdrop, the background discussion shifts to computer system memory organization.
Data in a given computer system is typically written and read in organized tiers of memory devices. These tiers are arranged according to the speed and volume with which data has to be written or read.
At one extreme of high speed and small volume, a Computer Processing Unit (CPU) employs on-chip cache registers and fast memory for storing small data units (multiple bytes) which move in and out of the CPU rapidly (sub-nanosecond speed).
The next lower tier involves programs and data that are stored in solid state memory (typically DRAM) utilized by the CPU and referenced in terms of the memory address space. This data is often accessed in a size of tens of bytes and at nanosecond speed.
In the mid-tier range, memory-mapped computer peripheral cards are found, where memory is tightly coupled to the CPU via onboard computer I/O buses such as PCI and PCI Express.
As utilization moves to the lower tiers, it involves mass data stored in electro-mechanical storage devices such as hard disk drives (HDDs). Disk arrays are often used, interconnected by parallel cables such as SCSI or by serial interfaces such as SATA. Since data is stored in a spinning magnetic storage medium, access speed is typically in milliseconds. The data is addressed in blocks of size exceeding one hundred bytes.
For very large storage requirements, arrays of distributed disk storage are often deployed. In the scenario of Direct Attached Storage (DAS), a short external cabled bus such as a SCSI or USB allows multiple hard disks to be located outside a computer.
In the scenario of Storage Area Network (SAN), such as a Fibre Channel network, a large number of hard drives may be distributed in multiple storage arrays, interconnected by local transmission links and switches and accessible by multiple clients. The clients of this mass storage access the storage server to retrieve data.
iSCSI is another example of a SAN application. In the case of iSCSI, data storage may be distributed over a wide area through a Wide Area Network (WAN). The Internet-SCSI (iSCSI) protocol encapsulates SCSI format data in Internet Protocol (IP) datagrams, which are then transported via the global Internet.
The lowest tier is utilized for storage and retrieval of larger data units such as files of Megabyte size at much lower speed (i.e. seconds). The Network File Server (NFS) is an example of a protocol for file retrieval over LANs and the Internet. Hard disks are the typical storage medium, but other slower speed medium such as magnetic tape may also be used. This very low tier of storage typically is used for archival purposes when huge volume of data is stored but retrieved very infrequently.
FIG. 2 shows a list of the various Tiers, arranged from highest performance to lowest performance, with Tier 0 being the highest performance. It may be observed, in reviewing the various tiers of memory, that the only type of memory access once you move out across the Ethernet network is block access or file access. The problem is there is presently no practical memory mapped access solution once you move beyond the host. Addressable memory has several advantages, including much finer granularity of data manipulation. With memory-mapped access, byte level manipulation and transactions are possible.
As 32-bit processors and operating systems give way to 64-bit systems, the associated memory map expands from 2^32=4 gigabyte of addressable memory space to 2^64=16 Exabyte of addressable memory space. Thus a tremendous amount of addressable memory is now possible. With this huge amount of memory potential available to the CPU, it is no longer technically necessary to assign mass storage to disk drives which limit the CPU to block or file level access. It would be desirable to have some portion of these memory-mapped resources distributed outside the computer and located in pools on a network or the Internet, such that the memory may be shared and addressable by multiple clients.
MeMAN disclosed in commonly assigned copending U.S. Patent Application Ser. No. 61/197,100 the teachings which are incorporated by reference is the concept of new classes—or “tiers”—of solid state addressable memory accessible via a high data rate Ethernet or the Internet. MeMAN stated another way, is the technology of addressable memory access via a network.
MeMAN enables the practical use of very large amounts of memory, external to a host computer system. With physical locality and confinement removed as an impediment, large quantities of memory, here before impractical to physically implement, now become practical. Memory chips and circuit cards (particularly in the form of solid state memory cards) no longer need be installed directly in a host system. Instead, the memory resources may be distributed or located centrally on a network, as convenient; in much the same manner that mass storage is presently implemented.
MeMAN leverages i-PCI as the foundational memory-mapped I/O expansion and virtualization protocol and extends the capability to include shared global memory resources. The net result is the potential for unprecedented amounts of collective memory—defined and managed in performance tiers—available for cooperative use between computer systems. MeMAN results in at least three new tiers of computer memory:
1. Memory-mapped computer memory located as Directly Attached Memory. This is located between Tiers 3 and 4 in FIG. 2.
2. Memory-mapped computer memory located on an Enterprise LAN. This is located between Tiers 6 and 7 in FIG. 2.
3. Memory-mapped computer memory located on the Internet. This is located between Tiers 9 and 10 in FIG. 2.
The resulting revised Memory Tiers are shown listed in FIG. 3.
Referring to the corresponding FIG. 4, it may be seen how all the Tiers, the remote I/O expansion chassis [104], and solid state memory cards [401] fit into to the overall memory scheme enabled by MeMAN.
Several technical challenges are presented by MeMAN, including how to handle the unprecedented amounts of collective memory and the related issues of simultaneous multiple host access and memory map sharing in a network environment.